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FOREWORD 



The effective dissemination of high-quality , research-based 
products is a Lundamental first step in the process of improving 
vocational education programs. Such dissemination not only 
eliminates the need to "rediscover the whtrel," but also saves in 
terms of product development costs. 

This report assumes a systematically developed, high-quality 
product based on research and development (RStD) efforts. Its 
focus is on activities of RStD product recipients. This report 
asks who they are, what they have done with the product, and what 
difference the product has made in their programs. 

Of necessity, the scope of the inquiry behind this report 
became increasingly restricted and narrower as the investigation 
moved from distribution to use and impact assessment . However, 
the report should help document the state of product dissem- 
ination in vocational education. In addition, it should help 
provide a database for designing an evaluation system for the 
outcomes of vocational education R&D, This design is discussed 
in a companion document titled System Design for Evaluating 
Vocational Education Research and Development , 

We acknowledge the assistance of David' P. Crandall, 
Executive Secretary of The Network, Inc., and DorenMadey 
Pinnell, Director of Corporate Relations, The Fuga School of 
Burliness, Duke University, who reviewed draft manuscripts of this 
report. Comments from M, Catherine Ashmore, Program Directc of 
Product Management for the National Center and Wesley E, Budke, 
Program Director for the National Center Clearinghouse, on the 
draft manuscript also were appreciated. 

This study of R&D product distribution, use, andimpact> 
could not havfe been completed without the cooperation and help of . 
state and local vocational educators. Persons completing the 
distribution records and responding to the telephone interviews 
were too numerous to list individually, but the West Virginia ^ 
educators deserve special mention for assisting with the impact 
study. In particular, I wish to express appreciation to Ed 
Suddath, State Department of Education, and Bill Jac9bson, 
Marshall University, They assisted in arranging for* the site 
visits, 'Local teachers participating in the study were: Mark 
Caldwell, Fayette Plateau Vo-Tech School; George Darling, 
Monongalia County Vo-Tech School; Chris Dunkle, Barboursville 
High Schodl; Homer Hall, Wheeling Park High School; Beth Monk-, 
Huntington East High School; Marianne Rastle, Charleston High 
School; Donna Snyder, Huntington East High School; and Barbara 
Spencer, Milton High School. 
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EXECUTIVE SUMMARY 



This study was undertaxen to document the distribution, ufee, 
and impact of vocational education research and development '(R&D) 
products developed with federal monies during fiscal years 1981 
and 1982/ R&D products developed under Programs of National 
Significance and ^state-adminiatered projects were reviewed. 

Three different types of data were collected to examine 
three different groups' of products. 

1. Distribution data came from products accepted into 
the Educational Resources Information Center (ERIC) 
during calendar year 1982. " These products were 
matched with entries in the Resources in Vocational 
Education (RIVE) database at the National Center for 
fiscal years 1981 and 1982 to verify the project 

source. ' 

* 

2. Use data came from recipients of R&D products 
selected for promotion by the National Center's ^ 
Dissemination and Utilization (D&U) Program. As 
these products were considered to be some of the very 
best available, these use data represent a "high 
water mark" for the utilization of vocational 
education R&D products. 

3. Impact data came from two products developed to 
increase users' competence in entrepreneurship . 
These products were a program planning guide and a 
set of eighteen instructional modules developed by 
the National Center for Research in Vocational 
Education. 

The methodology for collecting this information included 
several activities. First, requests for product distribution 
records were sent by mail to contact persons in each state. 
Twenty-two of the thirty-two states with products accepted by 
PJRIC responded with useable information on the role and 
organization of recipients. Next, usage information was obtained 
by telephone from fifty-five recipients of the D&U Program's 
selected products. Finally, impact data were obtained from six 
site locations in West ViL-ginia where the two entrepreneurship 
products were being used. 

. The findings from this study, summarized in the following, 
are discussed in greater detail in the report. 



Distribution Data 

o Of the sixty products produced under Programs of 

National Significance, a total of 19,396 copies were 
distributed during the calendar year in which the 
products were produced. 

^ , ix Q 



o Of the 248 products produced under state-administered 
, vocational education program improvement pro jects r a 
total of 56, 231 copies,^ were distributed during the 
calendar year in which the products were produced. 

o Over half (52 percent) *^of the product copies from 
Programs of National Significance were sent to state 
educat ion agencies . 

o Secondary schools represented the second largest 
audience for products produced under Program^ of 
National Significance (18 percent) • 

o Nearly half (48 pe,rcent ) of the staYe-administered 
product copies wer^ distrtibuted to secondary 
schools. , . , 

'o Two-year institutions represented a much larger 

audience for producbs produc€J,d under state' program 
improvement projects (20 percent) than they did for 
products produced under Programs of National 
Significance (3»6 percent). 

o Very few copies (.usually less than 10 percent of 
total copies) were recei^^e.d by nonprofit public and 
private orgcinizat ions , or by public agencies not 
connected with education* 

o Teachers received more R&D products than did other 
roles in secondary schools capturing 54 percent of 
. the copies distributed to school's by state agencies. 

o Products from Programs of National Significance with 
the most widespread dissemination were information 
papers and resource guides , products with the most 
widespread dissemination from state-administered 
projects were student guides and research/evaluation 
reports. 



Use Data 

The D and U selected product use study revealed the following 

Cab Seventy-five percent of the D&U selected product 

recipients remembered the product* Over half of this 
number r 63.6 percent ^ had used the product within the 
last year. 

o On the average ^ ^ arsons us ing the selected products 
with specit.:: groups each used the produ^cts with 
seventy- three teachers, 208 students , twenty -e ight 
administrators , forty-five counselors , and seven- 
teen advisory council members. 
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The following use data were collected on the entrepreneurship 
materials during the process of selecting them for the impact 
study* Participants in three nationwide workshops were asked 
telephone what they did with the products, 

o Ninety percent of the par t ic ipan ts in workshops 
promoting the entrepreneurship materials had used 
them for instructional purposes, 

o State education departments, in cooperation with 

universities r had used the entrepreneurship materials 
in workshops with 1^125 teachers, thirty-nine 
advisory council members, twenty-five directors of 
adult educat ion , and twenty- four community college 
deans, 

o Vocational-technical schools had used the entre-- 

preneurship modules with 325 adults, 1,184 students^ 
and sixteen instructors. 



The impact study of the entrepreneurship resource guides and 
eighteen curriculum modules yielded the following information 

o Four of the five school districts that had tried the 
entrepreneurship modules were conti nu ing to use them 
in place of materials they used previously with their 
senior classes. One comprehensive high school had 
initiated a new elective course on entrepreneurship 
based on the modules. 

The sixth school was continuing to use the materials 
in informal adult education courses, 

o Teachers had invested over 1,300 hours of non- 
reimbursed time preparing to use the modules, 

o Over 6,200 copies of pages within the modules have 
been dupl icatcd at local expense , 

o Approximately $2.00 in start-up costs per site were 
saved by using the entrepreneurship modules instead 
of purchasing new textbooks, 

o Entrepreneurship instruction accounted for 20,160 
: student contact hours annually in the six sites 

visi ted , 



Impact Data 




While gathering such impact data^ project staff also identified a 
number of factors that influence the ability of R&D materials to 
effect changes on a systematic basis. Some of these factors were 
as follows* 

o A school -state infrastructure .^ A good working 

relationship had to exist ampng local schools , the 
county office^ and the state education agency. 

o Interest in starting a business . Students and 

teachers who were most successful with the , . 

instruction came from families experienced in 
operating a business. 

o Linkages with the community * Speakers from the 

community were helpful in promoting entrepreneurship 
"and providing an opportunity for students, to see the 
management side of business. 

o Teacher competence . In addition to experience^ 
teachers needed training in how to teach entre- 
preneurship. Apparently^ the two-day workshop was 
adequate to start them learning on their own. 

Factors inhibiting changes also were identified — the most 
important of these being the inability of students to obtain 
loans, and take responsibility for starting and operating a 
business. 
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CHAPTER 1 
INTRODl'. 'ION 

Since 197U, over $350 million has been invested in 
vocational education program improvement projects (National 
Center for Research in Vocational Education 1980). In addition, 
data show that nearly $25 million v/as spent on state program 
improvement projects (research, exemplary, and curriculum 
development) in fiscal year 1981 alone (Budke 1983). Clearly, 
these rates of expenditure require accountability; yet to date, 
evidence of impact has been scant. 

Research evaluating the effectiveness of vocational 
education research and development is scarce for several reasons. 
First, evaluation of research and development product and 
information utilization and impact is expensive . Records and 
surveys may be used to measure relatively superficial impressions 
of impact, but measuring actual program changes usually is a 
complex process that often requires on-site interviews and 
observation. Second, appropriate evaluation takes time . 
Longitudinal studies are needed, but most fundir strategies are 
based upon yearly cycles. This makes long-term data collection 
designs nearly impossible to implement. Third, coordination of 
over fifty different state funding points is difficult. 
Conditions within each state dictate priorities for research and 
development, and the amount of money spent on R&D varies widely. 

These preconditions for measuring research and development 
impact add to the already heavy federal program improvement 
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burden. However , attention to program improvement is being urged 

by a number of nat ional--level agencies including the National 

Advisory Council on Vocational Education (1981). This council 

recommends that. . . "the federal role be refocused on clearly 

defined priorities to achieve access and equity and program 

improvement" (p. 3). But given recent reductions in federal 

spending, it is c'ifficult to justify an expensive evaluation 

system to assess educational research and development > 

effectiveness. Clearly a simplified and relatively low-cost 

system is needed for collecting data on the distribution, 

utilization, and impact of R&D information and products. 

This study examined outcomes achieved from the distribution, 

use, and impact of R&D information and products. Such a 

conceptual framework of investigation has proven beneficial to 

the development of impact assessment criteria (Hull, Adams, and 

Bragg 1983). Specifically, the objectives ^ this study embraced 

these three conceptual levels: 

o To determine the distribution of significant 
vocational education research and development 
products and information 

o To determine the use of significant vocational 
education research and development products and 
information 

o To determine the impact of significant vocational 
education research and covelopment products and 
information 

The data summaries obtained from this project will be used to 
develop and refine a system for evaluating vocational education 
R&D. 
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Procedures for collecting data at each of the three 
conceptual levels are described within each chapter. Thus it is 
sufficient at this point to mention only the geri'^ -a.l decision 
rules used to select "signif icantV^ vocational education R&D 
products • First, these products were developed from Programs of 
National Significance and state program improvement projects 
(funded under subpart: sections 131 research, 132 exemplary and 
innovative, and 133 curriculum development of P.L* 94-482, the 
Education Amendments of 1976/) Second, a particular year or 
years of products was selected from products accepted by the 
Educational Resource Information Center (ERIC). Distribution 
data were collected on all of these products. Use data came from 
products promoted by the National Center's Dissemination and 

Utilization Program* Impact data, because of the expense, were 

< 

collected on only t^Wo products that have been disseminated and 
used nationwide. 

This report is organized into three remaining chapters and a 
summary. Each of the chapters contain a description of data 
collection activities at each of the three conceptual levels: 
distribution, use, and impact. The three conceptual levels of 
distribution, use, and impact were taken from figure A-1 in 
appendix A. These levels lend themselves more readily to data 
collection following the distribution of products than do the 
other levels (e.g., product development and implementation) 



-^"Significant" is used to describe a group of products 
selected for study. Lower value or worth should not be ascribed 
to products excluded from this study. 
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in the conceptual framework. The "impact" level is described in 
the conceptual framework as the "effects" of product use. 

Different populations of products were selected for this 
study as a matter of convenience. For example^ products accepted 
into the ERIC database were used because they were worthy of 
widespread dissemination (a criterion in the model) and the names 
and ac'dresses of product distributors were readily available. 
'^Likewiser the twelve products promoted by the National Center's 
D&U Program were exemplary. In addition^ records of users were 
available both from product developers and from the D&U Program 
staff ♦ Finally r the two products on entrepreneurship were being 
used in locations convenient to the National Center? thus impact 
data were readily available. 

In summary, while the products selected for this study were 
not from the same population of products, they were all developed 
with federal vocational education funds. In addition, they were 
being used by practicing vocational educators. The samples 
appeared to be similar to products from prior studies and to each 
other. They represented the conceptual levels of impact very 
well. These levels are further explained and defined in appendix 
A. The conceptual framework in this appendix was used to guide 
the direction and conduct of this study, including the selection 
of the samples. 
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CHAPTER I.I 
DISTRIBUTION OF R&D PRODUCTS 

The measurement of product distribution (i.e., numbers of 
copies distributed) in this study was limited to the copies 
distributed during the year in which each product was released. 
Such a time limitation was necessary for comparison purposes 
because all of these products were entered in the ERIC system; 
thus there was no feasible way to measure the distribution of 
duplicated microfiche or print copies. 

Research coordinating unit (RCU) directors in states with 
products in the Research in Vocational Education (RIVE) database 
were the primary source of product distribution records for 
state-administered products. In some cases, it was necessary for 
RCU directors to obtain records from other persons in order to 
" provide the requested information. The National Center provided 
the distribution information for all but three of the products 
generated under Programs of National Significance, 

Selection of Products 
As mentioned previously, the information on product 
distribution came from research and development (R&D) products 
sponsored in whole or in part by the federal government. ' 
Products from state-administered program improvement projects (as 
funded by Sections 131, 132, and 133 of Public Law 94-482) and 
the corresponding Programs of National Significance, comprised 
the population of products for this particular study. This is 
the third study ( NCRVE 1980) and (Hull 1982), in nearly as many 
years to use this gtoeral population of products. 
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The products in the (iistr ibUt ion were similar to those used 
in the studies of use and impact. The products in the use study 

are described in detail in appendix C while descriptions of the 

i' 

products used in the impact study may be found in Chapter IV, A 
full range of product types (e.g.^ research reports through 
learn ing materials ) were included in the distribution* 

The sample was limited to those products entered into the 
RIVE database between the dates of January i, 1982 and December 
31 r 1982. In addition^ only products accepted by the Educational 
Resources Information Center (ERIC) or the Vocational Education 
Curriculum Materials (VECM) databases were used,,^ 

To obtain product distribution data^ a letter was addressed 
to the research coordinating unit director or a person on his or 
her staff. An attached sheet asked for information on the number 
of product recipients (see appendix B). The information was to 
come from available records , requiring no data collection by the 
recipient of the letter. A prior letter had been sent to the 
state director of vocational education in each state naming the 
person who would be asked for product distribution records. Two 
state directors provided substitute names for the state contact 
person. Telephone calls were made to contact persons who did not 
respond to the request. In some cases# it was possible to obtain 
the information over the telephone. 



2The sample included a total of 360 state-administered 
products. Some of these products entered in 1982 actually came 
from 1981 and 1982 projects. There were 105 products in the 
sample from 1982 projects, 242 products from 1983 projects, and 
13 products from the VECM database. All of the products from 
Programs of National Significance came from 1981 projects. 
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Telephone calls were made to most of the thirty-two states 
involved in this phase of the study. In fact, calls were made in 
advance of the letter requesting information if a state had seven 
or more products to report on. In addition, calls were made to 
persons in states who had not sent the information one week after 
it was due. In all, twenty-two states in the thirty-two state 
sample responded to the information request. 

Copies Distributed 

The primary statistic obtained from thi^. particular data 
analysis was the number of R&D product copies distributed to 
users. Primary users were categorized by role and organization 
for the state-administered, products and by organization for the 
products produced under programs of National Significance. Table 
1 shows the number of products captured by this sample » 

The telephone calls to the states probably accounted for the 
relatively high response rate (68.9 percent) for state-admin- 
istered products. This return rate compares favorably with the 
70 percent useable return rate in the Hull (1982) study and the 
68 percent rate from the NCRVE (1980) study. ThQ, Programs of 
National Significance product response rate was high because most 
of the records were available from the National Center. The 
average annual number of state-administered product copies 
distributed per product is 227. This is slightly lower than 1982 
study results of 280 average annual number of copies distributed 
per product. In the 1983 study, an average annual of 323 copies 
per product were distributed from federally administered Programs 
of National Significance. 
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' TABLE 1 

R&D PRODUCTS DISTRIBUTED BY PROGRAM SOURCE 



Source of 
Products^ 




PojSulation 
of .Products 


Number of 

Products with Use- 
arble Distribution 
Data 


, Number of 
Copies 
Distributed 


Average Number 
of Copies ' 
Per Product 


Programs of . 

National 

Significance 




> 

-60 


t 

60 


19,396^ ' 


323 


state- 
Administered 
Projects 




360 


248 


56,231^ 


227 



^This figure includes 1,006 copies distributed to national and international 
users. 21 

This figure includes 5,025 copies distributed to national and international 
users, not contained in any other tables. 



It should be noted that nearly half of the copies from 
state-administered products went to secondary schools (table 2). 
Most of these copies were disseminated to teachers. The next 
highest number'of copies went to secondary school administrators. 
This clearly indicates the most common way for R&D products to 
reach the classroom is through principals and/or teachers. Most 
of the other product recipient audiences look small by 
comparison. Administrators in all settings were key to reaching 
other staff, accounting for nearly half (47.3 percent) of the 
copies distributed. 

Table 3 portrays the distribution of federally administered 
products. The vast majority of copies went to state education 
agencies. This is consistent with the National Center's role m 
supporting state and national government agencies. The majority 
of products -produced trom Programs of National Significance are 
research and/or evaluation reports — materials that are most 
appropriate for policymakers. This product type category also 
was high among state-administered products. 

Table 3 contains information on the number of copies 
distributed from Programs of National Significance. These 
figures, categorized by type of product and organization of 
recipient, indicate that most of the products were either 
information papers or R&D reports. All of the products in the 
information papers category were produced by the National Center. 
The resource guide series contained three Resources in Vocational 
Education (RIVF.) volumes. Their distribution of 1,500 copies 
each inflated the copy numbers for this category from 748 to 
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TABLE 2 



TOTAL NUMBER OF STATE-ADMINISTERED PRODUCT 



1 


COPIES DISTRIBUTED BY 


ROLE AND 


ORGANIZATION 


OF USERS 






Organization 


Role 




Total 


Administrative 




/ 

Teachers 




Students 






N 


Percent 


N 


Percent 


N 


Percent 


State Education 


7,176 


• 5,051 


20.9 


1,691 


07.9 , 


434 


07.7 


Secondary School 


24,728 


8,362 


34.5 


13,396 


62.8 


2,970 


52.4 


Two -Year Injj^i^tution 


10,3i6 


3,659 


15.1 


4,570 


21.4 


2,087 


36.8 


University 


3,789 


2,584 


10.7 


1,047 


04.9 


158 


02.8 


Other Pviblic 
Organizations 


3,383 


« 

2., 967 


12.3 


398 


01.9 


18 


00.3 


Other Private 
Orgwizations 


1,814 


1,589 


06.6 


225 


01.1 


0 


00.0 


TOTAL 


51,206 


24,212 


100.1* 


21,327 


100.0 


5,667 


100.0 



^Percents do not add to 100.0 due to rounding 
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TABLE 3 



NUMBER OF FEDERALLY ADMINISTERED PRODUCT COPIES 
DISTRIBUTED TO ORGANIZATIONS BY TYPE OF PRODUCT 







Total 
Nuiuoeir or 
Copies 

IN — DU 

Products 




Type of Product 


i^pe or 
Organizations 




-■ 

Percent 


R&D 

Reports 
N = 23 
Products 


Information 
Papers 
N = 25 
Products 


Resource 
Guides and 
Proceedings 
N = 7 
Products 


Handouts 
and Training' 
Packages 
N = 5 
Products 


State Education 
Agencies 




10,106 


'".51.9 


649 


5,580 


' 3,110 


• 767 


Secondary Schools 




3,438 


17.6 


60 


2,191 


1,114 


73 


Two- Year 

Institutions 




708. 


3.6 


39 


62 


397 


210 


^Universities 




723 


3.6 


40 


195 


405 


83 


Other Public 
Organizations 




430 


2.1 


' 100 


123 


25 


, 182 


Other Private 
Organizations 




96 • 


> 

0.4 


16 


50 


16 


14 


Internal 

Distributions 




2,889 


14.8 


720 


1,777 


113 


279 


National^ Regional 
Distributions 




663 


4.3 


90 


467 


mm 


lUb 


International 
Distributions 




343 


1.7 


26 


201 


68 


„ 48 


Total 




19,396 




1,740 


10,646 


5,248 


1,7^.2 


Percent 






1%0.0 


9.0 


,54.8 


27.1 


9.1 
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5, 248, The average distribution for the -remaining four documents 
would have been 187 copies each without the RIVE volumes. The 
R&D products I except for tfiree products distributed through the 
National Center's "cost recovery system" # had relatively low 
distribution rates typically ranging f rom twenty--two to fifty-two 
copies each. (Most of these copies were sent only to the 
sponsor . ) 

Most of the copies (51.9 percent) from Programs of National 
Significance were distributed to state agencies such as research 
coordinating units. National and regional distributions were 
limited (4.3 percent of total copies) with the bulk of copies 
going to regional curriculum coordinating centers and national 
agencies such as professional associations. . Most of the 
secondary school distributions went to large urban school 
districts or to other organizations .through the National Center's 
cost-recovery sales,. Distributions to all other categories 
r*anged from 0;4 percent to 4.3 percent. Table 4 shows the 
breakdown for the distribution of state-administered products. 

The most frequent topic area of, the state-administered products 
was vocational education. New technology was the second most 
frequent subject in the reports. In descending order of 
frequency # the other areas were improvement of program 
management! spx^cial populations^ and economics. 

Most of the state-administered products were developed from 
oithpr curriculum development funds (section 133 of P.L. 94-482)i 
51.6 percent or research funds (section 131) # 35.5 percent. Only 
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TABLE 4 



NUMBER OF STATE-ADMINISTERED PRODUCTS 





DISTRIBUTED 


BY TYPE OF PRODUCT 






Type of Product 


Number 


of Products 


Number 


of Copies 




N 


Percent 


N 


Perc ..It 


Researca or evaluation 


102 


41. 1 


16,690 


32.6 


Tnf OT*in/5 ti on DaDGrs 
and resource guides 


9 


03.6 


1,633 


03.2 


' Handbooks 
» 


26 


09 .3 


8,372 


16 . 3 


Teacher guides 


13 


05.2 


1,354 


02.6 


Student guides 


98 


39 .5 


28,18^ 


55.0 


TOTAL 


248 




51,206 






1 — 
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12,9 percent of the products were funded from exemplary monies 
(section 132). Forty-seven percent of the products were released 
in July or August following the close of the June 30 fiscal year; 
so for nearly half of the products, dissemination records are for 
six months or less. 
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CHAPTER III 
D & U SELECTED PRODUCT USE 

The Dissemination and Utilization Program's (D&U) selected 
products represent some of the best available outputs from R&D 
projects nationwide. Most of these products have been developed 
with federal funds, and all have met the D&U Program's product 
selection criteria . 

The D&U Program is located in the Information Systems 
Division of the National Center for Research in Vocational 
Education. It promotes exemplary products by publicizing 
information about them and by recommending their use to resolve 
critical problems. It should be noted that the National Center 
does not distribute the actual products; it distributes only 
information about them. However, many of the D&U selected 
products have achieved nationwde distribution partly as a result 

of the information provided by the D&U Program^ .Detailed re cox.ds„ 

of D&U selected product requesters are kept, but to date there 
had been little opportunity to conduct follow-up studies to 
determine if requesters actually have usad the products. Thus 
twelve D&U selected products for calendar years 1981 and 1982 
were selected for this study. These particular years were 
selected because they provided a sufficient "publication" time to 
ensure that the products were in use. The D&U selected products 
are described^in figure 1. No research or evaluation reports 
were selected because th^se products are intended to initiate new 
programs in the field. In addition, most evaluation repo are 
site specific. Research reports of ten coi cd in fragmented 
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The High School Studant In The 
WORKING world: ' 
A Handbook for Counsilors 

Meets the need for a self-contained sourcebook for use with students 
entering "the world of work" during or immediatelv after high school. 
Working World gives indepth yet concise coverag-^ of the topics most 
crucial to secondary vocational counselors and placement officers: 

• Helping students explore their interests and potential 

• The career decision-making process 

• Involving the student's family 

• Getting hired and what to expect on the job 

• Exploring self-employment 

• Secondary and postsecondary vocational-technical training 
programs 

• Counseling the handicapped 

• planning a career guidance program 

• Sources of Information on present and future opportunities 

• Sources of free and Inexpensive counseling materials 

Includes worksheets and other aids to giving non-college-bound 
students the career planning help they need. Developed by the Texas 
Education Agencv- (1381/112 pp. ]• 



LEADERSHIP PRACTICES: 
For Directors of Vocational Education 

Serves as a professional handbook for directors of vociHtonul Hduc*^ 
tion, to assist them in creating conditions conducive to effective 
learning. Compiled from the contributions of vocationnl <'(lucators 
from the Division of Vocational Education, Ohio Department of 
Education, ( 1979/ 188 pp. 1 



A METHODOLOGY FOR 

READING SKILL IMPROVEMEf *^ IN 

VOCATIONAL SECONDARY PROGRAMS 

Assists vocational teachers in helping students read more effectively 
In vocational classes. Examples given of industrial arts activities can 
be readily adapted to other service areas. Developed by vocational 
education personnel in the Salem, Oregon, school district arid the 
Oregon State University School of Education. M981/60 pp.) 



IDEA BOCK: 
Meeting the Occupationat Informatton Needs 
of Disadvantaeed Youth 

Suggests innovative ways to deliver job market information to out-of- 
school youth-at a time when sound education and training decisions 
are more complex and more crucial than ever before. Strategies, 
models, and examples useful in delivering and restructuring career 
guidance services are presented in three chapters: 

f '.'DeJivery.of Occupational Informations-suggested program 

responses based on the experiences of selected projects and youth 

agencies 

• "Outreach Strategies" -guidelines and approaches for telling out- 
of -school youth about available Infornretion and services 

• "H ommendations for Planning"~-issues and conclusions regard- 
ing the major needs of youth and major program needs 

Developed by the Education and Work Program, Northwest Regional 
Educational Laboratory. (1981/56 pp. 1 



THE INTERDEPENDENT COMMUNITY: 
Collaborative Planning for Handlcappad Youth 

Describes an interagency collaborative planning process for improving 
career education services for handicapped youth. The intended out- 
come IS ;j detailed plan devised by members of various agencies and ^ 
supported by the community, parents, and employers. The Member's 
Guide \2'2 pp-l contains basic Information and planning exercises and 
is desKjnml to assist team members in reducing barriers to interagf?ncy 
cooperation. The Leader's Handbook (112 pp.1 contains the GuWa 
plus detailed rationales ond instructions tor team leaders. The collabo- 
rative plcjnnmg process can be readily adapted to any subject or pro- 
qr;hn area. (19801 



SOLAR CURRICULUM GUIDES 

Provides infornr^atFOn related to the preparation of solar energy tech- 
nicians through five courses on- 

• Applications of solar energy and future technology 

• Introduction to solar energy 

• Control theory and application 

• Collectors and energy storage 

• Solar heating and cooling systems 

Prepared by vocational educators it Seward County Community 
Cdllec)#, Kftrisas. t1980/260 ppi 



STATEWIDE OCCUPATIONAL EVALUATION SYSTEM 
ASSESSMENT PACKET 

Provides a unique mechanism for Initiating evaluation efforts or for 
clarifying, refining, and following thro< on state plans currently in 
place. This model builds a realistic base for program assessment and 
improvement on local expectations, and meshes those expectations 
, with broader objectives at the state l«veL The framework Is organized 
around key areas derived from- 

• The steps that characterize student progress through an occupa- 
tional ivogram (program awareness, program selection, develop- 
inont of basic and occupational skills, placement, etc.) 

• The administrative functions that support student progress 
'orogram and staff development, resources, and safety) 

Extensive outlines, sample forms, and worksheets in the assessment 
packet detail the process, ( 1981 /200 pp.1 A companion videotape. 
From Asseumant to Improvement, introduces the method. [31 min- 
utes, color) Developed by the University of the State of New York 
and the New York State Education Department, the set is equally 
applicable to program improvement efforts in other states. 



Figure 1. Description of D&U Products 
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SURVIVAL SKILLS FOR THE REAL WORLD 

Helps teachers equip students at the junior and senior high levels w»ih 
the practical skills needed to face the challenge of daily life and to 
achieve their personal goals. This curriculum guide focuses on iwo 
major areas: 

• Ernployment-locatint], applying for, interviewing for, and 
choosing a job; skills employers look for; and investigating job 
requirements 

• personal and family management-managing your money, respon- 
sibilities of car ownership, starting on your own, and rasponsibil- 
itiesas a tanpayer 

Surviviil Skills includes numerous activity sheets and handouts ready 
for duplication; a checklist to map student progress; a student list ol 
service organizations; and a teacher's section of additional print and 
audiovisual resources. Developed by the Oklahoma State Department 
of Education. (1981/192 pp. I 



VOCATIONAL EDUCATION DEMONSTRATIONS: 
Mmuring Imptct and Improving Eff«ctiv«n«ts 

Aids vocational educators in assessing and improving procedures used 
to plan, implement, evaluate, and monitor programs. This publication 
surveys, with examples and illustrations, procedures used to encourage 
adoption of successful practices and to select and fund demonstration 
projects. A glossary, a discussion of recommended criteria for review- 
ing propos'Ms, and a bibliography are included. (1980/136 ppj 



VOCATIONAL PLANNING GUIDE 

provides a practiOdl approach for local vocational administrators 
striving to plan eMoctive and efficient programs. Self-contained chap- 
ters permit planners to turn directly to the information that addresses 
their Immediate needs for- 

• Establishing vocational education's role within the overall LEA 
mission 

• Securing planning information by accessing available occupational 
data and collecting supplementary data 

• Accessing resources from both the public and private sectors 

• Acting on informat'un by synthesizing it in a systematic manner 

• Planning for legislative compliance in regard to representative 
planning input and other mandates 

• Establrshing working relationships with business, industry, labor, 
and other groups in the community . 

• Finding additional information and help through agencies, docu- 
ments, associations, and resource persons 

While geared primarily to Missouri personnel, the planning practices 
in thif guide will prove valuable to school, district, and community 
college planners in other states ?$ well. Developed by the Missouri 
Department of Elementary and Secondary Education. [1982/76 pp.] 



VOCATIONAL GUIDANCE: 
Planning Handbodk 

Helps in planning services for vocational education students. The 
Handbook, presents a model to help clarifv the meaning of vocational 
guidance and offers suggestions on basic services planner? nhould 
consider for their programs. Prepared by New Educational Directions. 
• Inc ■•; Grawf ordsviiie t indiana. 1979 / 98 pp. l- ^- 



VOCATIONAL EDUCATORS' HANDBOOK 
FOR ECONOMIC DEVELOPMENT 

Familiariees the novice business and industry liaison or coordinator 
With the basics of economic development and the role vocational 
education can logically and effectively assumt;. This *'how-to" manual 
will provide a springboard for customized, creative ideas a$ It focuses 
on two major themes: 

• Service to clients-the individual who wants to be trained for a 
productive and satisfying work life, as well as the business or 
industry needing labor trained to specjfic requirements 

• Teamwork -the public-private sector and agency/individual 
collaboration so crucial to assessing and meeting the needs of 
both client groups simultaneously 

The extent to which vocational education participates in local and 
state economic development remains largely up to the educators 
themselves. The Individual situation of the coordinator or linking 
agent taking this initiative will dictate whether to utilize the Hand- 
book in its entirety, or select among its five sections". 

• "The Role of Vocational Education tn Job Creation*' 

• "Preparing for Involvement in Economic Development" 
• •• "Plar>nmg industry Trainmg Programs" 

• "Sources of Funding) and Technical Assistnnce" 

• "Selected Readings and Bibliography" 

Seventeen case studies are summarized in an appendix as peitinent 
examples. Developed by the American Vocational Association. (1981/ 
141 pp.] 




Figure 1 Continued. 
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findings in need of further development before they can be used 
by others. 

The data collection had t:.hree purposes. First # it was 
intended to determine the extent of product use. Second^ it was 
intended to refine units of measure for product utilization 
assessment procedures. And finally^ it was intended to develop 
techniques for auditing records for product recipients. This 
report^ however, contains information relevant to the first 
purpose # product use# only. Findings related to the other two 
purposes will be incorporated into A System ^ for Evaluating 

Vocational Research and Development — the cn mion document to 
be developed from this study. 

As mentioned previously, the highly selective manner in 
which the D&U products were chosen, demands that results must be 
viewed as a "high water mark" in regard to expectations for 
vocational education R&D products. 

Select ion of Respondents 
Lists of probable recipients of these D&U products were 
compiled from the National Center's "Field Activity Record" form 
and from files of product developers and distributors. Some 
selectivity occurred when the lists were compiled, but an attempt 
was made to select recipients randomly. There was a tendency to 
select distribution listings from purchase orders ^nd letter 
requests that included identifiable names of use^rs. The names 
came from 1981 and 1.982. distribution lists for the products^ 

Zero to sixteen users names were available for the twelve 
products. The requester respondents were njt limited to 
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the states where the product was developed. In fact, over 80 
percent of the names came from lists of nationwide distributions 
of the product • 

Table 5 shows the actual number of persons contacted about 
the D&U products • This does not include persons on the 
distribution list who were out-of-town at the time of contact, 
who had moved, or otherwise were not available. To comply with 
the minimum exclusion requirements for FEDAC clearance, not more 
than nine persons weire contacted for each of the products. 

A total of seventy-three persons were interviewed on eleven 
products* Individuals^, who did not remember the product; or who 
had passed the product along to others without looking at it, 
were counted as not having any information about the product* 
Those who had examined it but decided against using it were 
counted as a useable response* There were eight "no use" 
responses from the fifty--five persons who remembered the product* 
Their reasons for ^ot using the product varied, ranging from 
"they had received it too recently" to "they were waiting for 
staff changes to introduce the product." The percentage column 
is the percent of useable responses from the persons contacted* • 
It was not possible to determine exactly how many persons had 
received the product or the total number of products distributed* 

Analys i s of Results 
Telephone calls were made by project staff to product 
recipients* The findings from the telephone calls were 
summarized as scenarios on a product-by-product basis* Some 
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TABLE 5 

NUMBER OF PRODUCT ySE RESPONDENTS 
BY PRODUCT TITLE 





Persons 

Contacted from 


Persons 
with no 


Useable Responses 


Product Title 


Distribution 
List 


Information 
on Product 


Number 


Percentage 


World: A Handbook for Counselors 


7 




7 


100% 


Idea Book: Meeting the Occupational 

J> 1 « i. V^JL iim U Xwi * JlNCCCiO X OC&U V CI A 1 l«C&>^ 

Youth 

c 


9 


3 


O 


'5 ft 


The Interdependent Community: Collabor- 
ative Planning for Handicapped Youth 


8 


5 


3 


38% 


Leadership Practices: For Directors 
of Vocational Education 


8 


2 


6 


75% 


A Methodology for Reading Skill 

Improvement in Vocational Secondary 
Programs 


6 




6 


100% 


Solar Curriculum Guides 




1 


3 


75% 


statewide Occupational Evaluation 
System Assessment Packet 


2 




2 


100% 



3f 
D 

ERIC 



TABLE 5 Continued. 





Persons 

Contacted from 


Persons 
with no 




Product Title 


Distribution 
List 


Information 
on Product 


Number 


Percentage 


Survival Skills for 'the Real VJorld 


5 


1 


4 


SO 


Vocational Education Demonstrations: 
Measuring Impact and Improving 
Ef fect\veness 


9 


5 


A 
H 


AA 


Vocational Educators* Handbook for 
Economic Development 


None 






mm 


Vocational Guidance: Planning Handbook 


8 




8 


100 


Vocational Planning Guide 


7 


1 


6 


86 


TOTAL 


73 


18 


55 


75 
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structure for the interviews was provided through the use of 
"general comment" headings that were common to each product 
summary, Howeverr the information in the scenarios varied widely 
due to the open-ended nature of the telephone interviews^ Thus 
even though a composite picture of product use emerges within 
^'each scenario, most of this information cannot be aggregated' 
across products because of the differences in the products and in 
the roles and organizations of the product uFcrs^ 

Some of the more quantifiable statements from the interviews 
are summarized in table 6, Obtaining these statements required 
classifying the D&U products into categories of product types and 
categorizing the respondent comments into three headings: 
personal use^^use with others, or programmatic/organizational 



interviewers* None of the DfitU selected products wer^ classified 
as research reports , policy papers, position papers, or student 
materials. Therefore, these categories were omitted from table 
6. 

The information in table 6 shows "personal use" of the D&U 
selected products to be the most common, response, "Use with 
others" was the next most common response .with a surprisingly 



strong showing in the programmatic/organizational use for 
handbooks and procedural guides. The reader should remember that 
these percentages reflect what the interviewee voluntarily 
mentioned, as us e_^^^ Memory and the length of 

interview time trendnd to depress the frequency of uses reported. 



use. These classification judgments were made by the 
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TABLE 6 



PRODUCT USE SUMMARY 









Types of Products 


Type of 


Guides r Directories « 


Handbooks , 


Teacher 


Use 


Bibliographies^ 




Procedural 


Guides 






Literature Reviews 


VJ U X U V o 




N ■ 7 


f 




N «• 16 




K « 32 


• 








Percentage * 


Conuoent 


Percentage^ 


Comnent 






Perffonal Uae^ 


43.8 


Read or scanned 


8].« 3 


Read or scanned 


85,7 


Read or scannc 




50.0 




12.5 


Personal pref- 






















12.5 


Adapted product 










o 




for own use 










A* 
















12.5 


Conducted research 










f 

use wiuii uuiiQjrs 


25.0 


Recommended as 


31*3 


Involved dtaff 


14*3 


Shared with 






resource 








university 














faculty 




6.3 


Advisory Council 


15,6 


Advisory Committed 


42.8 


Teachers and 














students 




31,3 


Administrators 


3,1 


Described in 














newsletter 








12,5 


Placed in a library 


3.1 


Distributed to 














administrator 






Progreuomatic/ 


6.3 


Developed into a 


3*1 


Developed sef. of 






Organizational use 




curriculum plan 




standards 








12.5 


Policy decisions 


40.6 


Imoroved 














instruction 


1 





Percentages were computed as a proportion of the N in each column* The N is a composite of the 
useable responses from proudcta classified in this colxamn. 
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TABLE 6 ContinuecU 
PRODUCT USE SUMMARY 





Types of Products H 


Guides, Directorieo/ 
Bibliographies, 
Literature Reviews 
N - 16 


Handbooks 
Procedura 
Guides 
N - 32 


1 


Teacher 
Guides 
N « 7 




Programatic/ 
Or9anizationaZ Use 


percentage 


Ccnment 


^ Percentage* 


Coement 


Percentage 


Ccnnent 






6.2 

3*1 
6.2 
6.2 


Conduct state** 
wide and nation- 
worki hops 

Developed 

handbook 

Regional planning 
model 

Conducted needs 
assessment 






-percentages were computed as « proportion of the N in each column. The N is a con,,osite of the 
useable reaponsea from products classified in this column, 

41 



erIc 



For example, as thw. interview lengthened, answers became 
shorter. Also, some respondents could not remember events that 
had occurred more than one or two years ago. 



Chapter IV 
ASSESSMEN'i" OF PRODUCT IMPACT 
An investigative study of product impact was conducted by 
project staff in October, 1983. One exemplary product entitled 
Entrepreneurship in Vocational Education was aelected for study 
t^nm the produc promoted by the Dissemination and Utilization 
(D&U) Program, In addition, a product developed by the National 
Center entitled A Program for Acquiring Competence in 
Entrepreneurship (PACE), also was studied. Site visits were made 
to six communities in West Virginia where the products had been 
used. Observations from these communities were summarized and 
analyzed across sites as well as within sites. The results were 
reported along dimensions consistent with the' impact framework 
developed in a companion document, Hull and Adams (forthcoming), 

• 

Selection of the Product 
The products promoted by the D&U Program during 1981 and 

1982 were reviewed by project staff. Selection was based on the 

following criteria: 

o Widespread distribution of the product 

o Similarity of the product to the population of 
products in that theme area 

o Use of the product in several geographically adjacent 
areas 

The substantive theme of "entrepreneurship" was selected because 
of widespread interest in the area and the potential for 
expansion in vocational education. 
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A program planning guide entitled Entrepreneurship in 
" Vocational Education was not only available but also repre- 
sentative of products on entrepreneurship. .This guide listed 
five steps for developing entrepreneurship training programs, and 
reviewed eight curriculum products. Selected details of each 
product were copipared in two easy-to-read charts. The PACK 
modui s, eighteen in all, were produced as one title per module. 
They were written at three different reading levels. Available 
with the modules was an Instructor's Guide with objectives for 
each unit, transparencies, and assessment questions answers. 

The PACE modules and the program planning guide had been 
publicized and promoted through a series of workshops. The 
Entrepreneurship in Vocat i onal Education document was distributed 
at four different National Academy workshops, noted below: 
o St, Louis, MO December 198 2 

o Raleigh, NC October 1982 

o Marquette, MI March 1983 

o Columbus, OH October 1983 

The planning guide and the moduler. were discussed at four 
regional meetings of distributive education teacher educators and 

state staffs at the following locations: 

o Springfield, MA September 1981 

o Nashville, TN September 1981 

o Milwaukee, WI October 1981 

o Las Vegas, NE November 1981 
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The modules were distributed to several West Virginia 
communities in preparation for prepubl icat ion revisions. 
Teachers present at a North Bend State Park meeting in November 
1982 were asked to test the units and recommend revisions, Tho 
meeting followed an earlier meeting at Morgantown in August 1982# 

Inservice workshops for repres.entat ives of sch ol districts 
and state education agencies interested in entrepreneurship have 
been conducted in Nebraska and Wisconsin, "N^w Product" sessions 
at the American Vocational Association convention and. promotional 
activities through the state liaison representatives (SLRs) have 
spurred product sales. From July 1980 through June 1983.; the 
National Center sold 758 copies of the program planning guide in 
entrepreneurship and 9,846 PACE modules. 

The persons participating in the National Academy workshops 
were interviewed to determine if the products were being used. 
This was completed prior to making a commitment for field si^te 
visits in West Virginia, The results of the interviews are 
summarized in the next section of this report, 

Implementat ion and Use 
Although the time frame for follow-up of products presented 
to 1982 and 1983 workshop participants was a little short, it was 
believed participants would have had time to use the products 
within this period. The participants at the North Bend State 
Park meeting had received a copy of the program planning guide 
during earlier pile tests and a set of PACE modules at the 
meeting. Some persons at other locations had purchased PACE 




modules on their own. The following statements summarize how 



some of the participants were using the program planning guide 
and PACE. Only 10 percent of the persons contacted had not ur 
the materials. 

f 

o A professor in Virginia was using PACR with other 
resources to construct a teacher resource guide on 
entrepreneurship for statewide distribution. 

o An inservice session was conducted with sixteen 
vocational instructors in an intermediate school 
district in Michigan using handouts from the program 
planning guide. 

o A semester course in entrepreneurship is being 

developed in North Carolina for implementation by the 
state office? in addition, 310 distributive education 
teachers were informed about PACE via a newsletter. 

o Multiple copies of the program planning guide on 
entrepreneurship were distributed to cooperative 
education teachers in the Houston Independent School 
District. 

o A community college instructor in Michigan used 

handouts from the program planning guide in manage- 
ment classes of thirty-three and forty-four 
students . 

o The research coordinating unit director in Wisconsin 
conducted an inservice (one day each) with sixteen 
intermediate districts and five postsecondary 
districts using these materials and others on 
entrepreneurship? average attendance was forty-five 
persons per , session. 

o An evening instructor at a community college in 

Michigan used PACE with an adult class on "Starting a 
Business." Thirty students were enrolled. 

o A research and development project director in 
Oklahoma shared the information in the program 
planning guide with staff. It was then used as a 
resource to prepare for a national workshop and 
twenty regional workshops. 

o A member of a state advisory council in Alabama 
shared the information with the thirty-nine-member 
council and the distributive education state 
supervisor. 



o 



A university professor in Missouri used the program 
planning guide as a regular resource in his classes 
on retailing. 



o A high school teacher in Ohio used the PACE modules 
with thirty senior students in distributive 
education. 

o The revised PACE is being made available to high 
school teachers statewide through a "Center for 
Business Teachers" in California. 

o A distributive education teacher in a county' 

vocational-technical center in Ohio used PACE with 
two seniors last year. 

o A university professor in Pennsylvania was using the 
planning guide with classes in farm management. 

o The program planning guide was used in four work^^hops 
with high school students in Maryland. An average of 
forty-five students attended each workshop. 

o PACE is being made available to local school dis- 
tricts through an inter-library loan system in 
Colorado. 

o The program planning guide is being used by teacher 
educators in adult education classes at a state 
teachers'^ college in Kansas. " 

o A technical institute in Ohio used PACE in business 
education classes with an average attendance of 
twenty-six studenta per class. 

o A high school teacher in Maryland used PACE with 
seventeen second-year distributive educat ion 
students, and shared the materials with her advisory 
counc il * 

o A "Center for Small Businesses" in Florida used PACE 
with 150 adult students. 

o A joint vocational school in Ohio used PACE with 
15U students in a summer workshop. 

() A university professor in counselor education in 
North Carolina used the program planning guide with 
fifty students. 

o A regional service agency in Maryland used the pro- 
gram planning guide as an inservice with five high 
school vocational teachers. 
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o A community college dean in Alaska used the 

program planning guide with his staff in business. 

o A university professor in Missouri used the planning 
guide to inform faculty in her department of entre- 
preneurship as; a priority in education. 

o A restaurant association education director in New 
Mexico used the program' planning guide , in designing 
an "Introduction to Entrepreneurship" guide for 
statewide distr ibut ion to high school food service 
programs and Distributive Education Clubs of 
America Chapters. 

o A small business training coordinator in Oklahoma 
used PACE with 1^000 students in a vocational tech- 
nical school 9 with 175 adults in a seminar , and with 
three co-workers. 

o A specialist at the Office of Advocacy in the U.S, 
Small Business Administration shared PACE with 
three supervisors and twenty-five people in a 
reg ional workshop. 

o A postsecohdary and adult education specialist in the 
Maryland IDepartment of Education used the program 
planning guide in separate workshops with twenty-five 
directors of adult education^ four statewide 
curriculum conferences ^ and twenty-four deans of 
community colleges • 



The preceding examples of product use are isolated and dis- 
jointedi but they indicate what is reasonable to expect from the 
dissemination jOf R&D products* Nationwide implementation of R&O 
products depends heavily on a network of people^ sometimes in 
specialized roles/ linked together through communication and 
informal relationships. The preceding description came from 
people linked to one another only through workshops on entre- 
preneurship. They were self-selected for the workshops; no 
attempt was made to interrelate their activities. 
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R&D Product Impact 

Education is a state responsibility in this country, so it 
is logical to assess impact on a state-by-state basis. Tn this 
case, West Virginia was selected for site visits because one of 
the products, Entrepreneurship in Vocational Education , had been 
developed by state staff members, ^ The other product, ^ 
Program for Acquiring Competence in Entrepreneurship ( PACE ) , had 
been field tested in West Virginia. State leaders were receptive 
to assessing impact and willing to contribute some of their time. 
Travel costs were held down and disruption minimized by limiting 
the observations to half-day visits. 

Interviews were conducted during the weeks of October 3 and 
October 10, 1983 in the following sites: Charleston, Huntington, 
Milton, Morgantown, Oak Hill, and Wheeling. Before discussing 
the findings of the impact study, however, a word needs to be 
said about the evaluation of substantive concept of "entre- 
preneurship." 

"Entrepreneurship" as a concept to be taught has varied 

meanings in different contexts. For example, it is difficult to 

aggregate data or comments across the following notions: 

o Instruction on starting a small business is equated, 
by some people, with instruction on entrepreneurship. 

o Entrepreneurship is defined by some as a vertical 
dimension of the career development process, but is 
listed in the vocational education data system 
(VEDS) as a single, competency-based component at 
one level. 



3since development of the guide, both staff members have 
moved to other positions. 
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o Other persons insist on the taking of initiative, 
often associated with some risks, before a person 
can be considered an entrepreneur • 

If en trepreneurship implies a commitment to establish a 
business enterprise, then achievement of success depends on many 
factors that are often not present when a person enrolls in a 
class. Among these factors are the following: 

o Availability of risk capital 

o Des ire to be independent 

o Knowledge of a product market 

> 

o Willingness to be opportunistic 
Most high school students do not possess these qualifi- 
cations. Criteria for judging the effectiveness of classes in 
entrepreneurship therefore, must take these special con- 
siderations into account. Time is required for individuals to 
decide to start a business. The opportunity to start one may 
come yearr ^fter enrolling in a class on entrepreneurship. This 
makes the evaluation of entrepreneurship education effects on 
students within a defined time frame very difficult. 

Impact assessment of recent entrepreneurship education 
activities is exacerbated by another factor. High quality 
distributive education materials are available for secondary 
education students. These distributive education materials are 
being used to prepare students for .he world of work. However, 
the emphasis on working for oneself in these materials, rather 
than working for other jDeople, needs tn be reinforced. Many 
rcttourcoF; are available to vocntional educators who want to place 
more emphasis on entrepreneurship. 
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Consistent with the impact assessment model (included in 
appendix A), the impact criteria used here focused on indicators 
of personal satisfaction (or dissatisfaction) with use of the 
materials, individual growth, organizational change, and societal 
contributions., Although en t repreneur ship educational materials 
have the potential for influencing indicators of economic 
productive .y such' as the number of businesses started by users, 
insufficienc time had elapsed for this to take place. 
Realistically, the research team did not expect to find much 
evidence of societal contributions. 

The six sites visited had obtained the PACE units in 
November 1982, so this assessment followed receipt of the units 
by nearly a year. The program planning guide had also been 
shared with recipients some time earlier, but its use was 
reinforced at the November 1982 conference at North Bend State 
Park. The six sites visited included two county vocational 
schools and four comprehensive high schools. In almost every 
case the principal recipient of the materials was a distributive 
education teacher; the sole exception was a teacher at a 
vocational school with a Diversified Occupations Program. All of 
the teachers supervised and/or were responsible for placement of 
students in jobs in the community. 

A primary difference in the settings was the amount of time 
devoted to class instruction. The typical vocational school 
class had three, sixty-minute periods in the atternoon or morning 
for instruction, for a tota^ of 180 minutes per day available for 
using the product. The comprehensive high school had one fifty- 
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minute period per day tor product use time. The effect of these 

differences was to give the vocational school student more 

opportunity to study entrepreneurship in greater depth. This 

added time was especially helpful in the completion of individual 

plans for starting small businesses. 

Use of the PACE units varied greatly from site to site. At 

one of the county vocational schools, Level 3 materials were used 

informally with adults. The local chamber of commerce referred 

adults interested in starting a business to the local teacher. 

An important consideration in this referral was the urgency of 

the need for advice. Sometimes the prospective business person 

did not want to wait until a representative of the U.S. Small 

Business Administration could conduct a short course on the 

topic. Also, the reputation of the teacher was a factor. Local 

people seemed to have confidence in his advice. Some of the 

students he supervised were placed in businesses where the 

prospective entrepreneurs worked. The following outcomes were - 

identified at the site: ' 

o Four small businesses were started in the community: 
a lawn and tree service, a home answering service, 
a gourmet shop in the downtown business district, 
and a confectionery business. 

o Advice on starting a small, business was more 
access ible . 

The other county vocational school used the Level 2 PACE 
units with a class of twelve seniors in the spring of 1983. The 
materials were also being used this fall with fifteen seniors. 
About half of the seniors this fall were working in jobs where 
they could see entrepreneurship from the employer's point of 
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view. The teacher credits PACH with the eoUowing student 
outcomes ; 



o The acquisition of more information about entrepre^ 
nuership in less time 

o A better understanding of how to operate a t)\!isiness-- ^ 

o An increased appreciation by most students of 

the need to go on in school ^ \ " 

At the teacher levels the effects were: 
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o A satisfaction with the two-day training program 

o Ease of integration of material with other 
references 

o An Increased awareness of the need for a unit 

on bookkeeping (not previously taught); this unit 
will be taught via an accounting program recently 
available on a microcomputer 

6 Plans to offer, an adult class during the spring of 
1984 on. "Beginning a Small Business" 

Some school districts are presently experiencing a loss in 
■enrollment. The possibility of attracting students to marketing 
b'y offering a course in entrepreneurship was mentioned by more 
than one teacher. Offering an elective course on "entre- 
preneurship could make better use of a teacher's expertise rather 
than assigning him/her to an alternative class in an area 
unfamiliar to the teacher. Th^ PACE units could help facilitate 
the offering. 

Among the five comprehensive high school programs studied, 
one was offering a brand new course on entrepreneurship as an 
elective? thret were using the units with second-year classes in 
distributive education; and one had used the PACE units with 
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adults. Most of the twenty-one enrollees in the adult class were 
paid to be there with Comprehensive Education and Training Act 
(CETA) funds. The class met for seventeen weeks in the jprinq of 
1983 for six hours^ five days a week during regular school hours. 

The coordinator of the class was highly positive about the 
PACE units^ but the instructor and students were not available 
for comment. Five people in the class reportedly were interested 
in opening a business; but for many^ the class appeared to be 
part of an requirement for staying in an income maintenance 
program. A factory had closed in the area^ and students were 
paid to attend class. There were no other effects at that site; 
the entrepreneurship materials were not being used with secondary^ 
school students. ^ 

The entrepreneurship materials appeared to be meeting a need 
in the remaining high school distributive education programs. 
Teachers started using the modules for the senior year curriculum 
beginning in the fall of 1983. This was facilitated by a 
decision at the county level to duplicate sufficient copies of 
the units for each student to have a persorij^l study copy. These 
teachers had Units 6-18; for some reason unknown to the teachers, 
Units 1-5 had not been duplicated. 

One teacher had initiated a new course in entrepreneurship. 
It took a year for her to complete the certification process. 
She had to meet the state office of education code classifi- 
Crition, develop a course description, and obtain her principal's 
endorsement before submitting the proposed description to the 
county school board for approval. Twenty-two students had 
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enrolled in the class; the teacher was highly pleased with the 
selection of students enrolled. She credits the guidance 
counselors as the main reason for appropriate student enrollment. 

Collectively, a great deal of effort was put forth by 
educators in implementing and using the program planniag guide 
and the PACE modules. Table 7 shows the amount of time invested > 
in preparing for and instructing these classes in entrepre- 
neurship. Preparation tine includes two days of workshop . 
activity (from five to eight hours per week) and in some cases up 
to fifty hours of preparation time in the summer. The 
instructional time is basejj on class schedules. As mentioned 
previously, the vocational schools had 180 minutes of tip;e 
available per day, while comprehensive high schools had fifty 
minutes. Average enrollments did not differ significantly. 

As noted in Table 8, comprehensive high schools enrolled 
slightly more students per class. The reader should keep in mind 
that these figures are based on very small "N" values — only six 
sites. Nevertheless, the numbers are consistent across schools. 
The number of students with their own businesses, for example, 
were two in each instance. These student businesses tended to be 
businesses operated by the student as part of their parent's 
companies. Rarely would a teacher expect to find high school 
students owning and operating businesses independent of their 
parents . 

The effects of entrepreneursh ip module use in six secondary 
high schools are as follows: 

o Initiation of a new course entitled " Kntrepreneurship" 
in one comprehensive hi(jh school 
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TABLE 7 



AVERAGE NUMBER OF HOURS SPENT BY INSTRUCTORS 
PREPARING AND TEACHING ENTREPRENEURSHIP BY YEAR 



Type of 


Preparation 


Instructional 


School 


Time 


Time 


Vocational School 


252 


5^.0 


Comprehensive 


262 


360 


* High School 







TABLE 8 

I 

AVERAGE NUMBER OF STUDENTS 
SERVED BY ENTREPRENEURSHIP 



Type of 




Class 


Students with 


School 




Enrollment 


Their own Business 


Vocational School 




14 


2 


Comprehensive 




21 


2 


High School 
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o Investment of over 1,300 hours of nonreimbursed 
teacher preparation t ime, among f ive s ites 

o Dupl icat ion of over 6,200 pages of in struct iond 1 
material at l6cal expense 

o A total of apporox imagely 20,160 annual student 
contact hours in the six sites visited 

o Approximately $200 in start-up costs per site were 
saved by using the entrepreneurship materials rather 
than purchasing new textbooks 

o Creation of a library of papers on "Great Entre- 
preneurs" 

o Students tended to empathize with problems of" bus- 
iness managers 

o Student vocabulary changed; a "paper" became an 
"invoice" 

o County wide informal voluntary get-togethers among 
teachers and entrepreneurship students to share 
ideas and experience 

o Parents were supportive of classes on entre- 
preneuship 

o Course descriptions were changed to accomodate 
entrepreneurship themes 

No negative effects were noted. Undoubtedly, other material 
was supplanted by the entrepreneurship instruction? but the 
teachers uniformly saw this as an improvement. Many factors 
^ existed to facilitate and/or inhibit the impact of 
entrepreneurship materials. Among them were the following: 

^' acilitating Factors 

o School -St ate Infrastructure . A variety of conditions 
came together to support the introduction of entre- 
preneurship materials in West Virginia. Authors of 
the program planning guide were employed at the state 
level; they had been local tdachers of Distributive 
Education. Local teachers knew and respected them as 
leaders. Cooperation between the local and county 
levels was documented in this case study by the 
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duplication of the PACE materials. Witnin the 
schools^ relationships were important; principals had 
to endorse changes in course content. 

o Background of Participants . Enthusiastic study of 
entrepreneurs hip could be 1 inked to personal exper-. 
iences of both teachers and students . Often teachers 
had engaged in business themselves either as entre- 
preneurs or as employees. Students interested in 
start ing the ir own busi nesses tended to be sons and 
daughters of proprietors. 

o Linkages to the Community ^ In the broadest sense ^ 
the entrepreneurship units provided an opportunity 
for community leaders in business and industry to 
share their expertise with students. All of the 
field sites had speakers^ and^'other kinds of involve- 
ment from the community. Of particular importance 
were the ties to the local chambers of commerce and 
the U.S. Small Business Administration. An oppor- 
tunity also existed for secondary school teachers to 
work together with other governmental agencies to 
serve the needs of adults in the community. 

o Teacher Competence . Two aspects of entrepreneurs^ 
relate to competence of the teachers. First r all 
were distributive education teachers or former 
distributive education teachers. They knew business 
people in the community in some cases they had 
operated businesses of tneir own. Teaching entre- 
preneurship was an extens ion of what they had been 
doing.. Secondly, the two-day workshop served to^ 
review the essential information. They felt 
"comfortable" with the materials despite the lack of 
exper ience actual ly using them. 



Inhibit ing Factors 

o Strict Certification Process . The strict process for 
certifying new courses tended to slow down the implex- 
mentation of a new course* It took nearly a year 
from the time the materials were distributed until 
the course could be offered. While this is not ex- 
cessiver it is a time delay that must be considered 
when planning innovations in a curriculum. 

Teacher inexperience . All but one of the teachers in 
one of the counties were first- or second-year 
teachers. The burden of trying something new was 
added to the task of preparing instruction for the 
first time. New teacher use of the modules was not as 
effective as use by experienced teachers. 
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o Qualifications of the Students ^ If one accepts the 
strict definition of an entrepreneur as one who 
"starts a business/' then most of the students in the 
classes studied could not be considered qualified as 
potential entrepreneurs* They would not be good 
risks for venture capital. Secondary school students 
lack business experience, and most of the CETA- 
sponsored adults lack the funds to start their own 
business* Selection of students appears to be a key 
factor in the potential success of a program. 

o Limited Parent Involvement . With the notable ex- 
ception of a few parents who owned their own bus- 
iness r most parents were only mildly interested in 
the content of school instruction . Few parents 
visited classrooms during orientation programs; and 
teachers received few questions from parents. 



Some circumstantial factors seemed to exert conflicting 
influences on the success of the entrepreneurship program. For 
example, the economic recession both stimulated interest in 
enrollment among adults as a means of making money and decreased 
the availability of capital to start a business. 

The effect of the PACE modules and the program planning 
guide on entrepreneurship instruction seemed to be positive. The 
teachers were excited and pleased with their accomplishments. 
The students were interested and also a- bit intrigued by the 
possibility of starting their own business. 



.3 59 



SUMMARY 



Having examined the records ot product distribution, 



" f ol lowed-up" recipients of D&IJ selected products^ and 



interviewed users of R&D products in six sites^ what can be said 
of R&D product effectiveness? Can it be documented? And 
furthermore^ what is likely to be the substantive outcomes from 
disseminating R&D products? 

Clearly r the answer to, the question of documentation is a 
resounding "yes," Evaluative information can be gathered on any 
or all R&D products if money and time is available to pay for 



this activity. The problem is that the benefits from collecting 
this informc:wion may not outweigh the cost. Relatively low-cost 
information on product distribution can be collecte^^ from 
producers and distributors of thes^ products. In this study 
distribution data were collected on 248 products for a 68.8 
percent return of useable record sheets. This return of 
state-administered products has been consistently near this 
percentage for three years. The quality of information has also 
been consistent over a three^-year period. At the minimum^ 
governmental agencies should know how many products have been 
produced from an expenditure of taxpayers' dollars and where 



copies of proxiucts have been sent. 

Knowing what recipients do v;ith the products presents a 
different type of problem. While questionnaires can delineate 
categories r product use, readers of iT't^ports may not gain the 
flavor of adaptations in the product, circumstances surrounding 
its usor and implications from that use. Initial responses to 
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telephone inquiries about product use bad to bo probed with other 
questions. The product recipient often did not recognize all of 
the dimensions of use at first glance. It should be possible to 
obtain relatively superficial information about use from a 
written survey document; but deeper understandings will have to 
come through personal interviews either by telephone or faco- 
to-face. This "personal touch", however, pushes the cost up j 
thus governmental agencies may have to consider seriously the 
value of the information prior to launching an investigation into 
product use. In this study product use data was obtained from 
users of eleven of twelve products being reviewed. A feasibility 
study of costs and possible benefits of product use and impact 
data is always in order. 

The substantive results from this study of D&U product use 
painted a rather positive picture. Very few people reported "no 
use" of the product they had requested. Those reporting no use 
had a reason, such as a limited amount of elapsed time since 
receiving the product, lack of resources, and so forth. The 
importance of elapsed time was reinforced by the impact 
assessment. It took a full year for a new course to be 
constructed and approved following the introduction of 
instructional units in the fall. Results from the introduction 
of R&D products into classrooms come slowly — particularly in 
areas such as entrepreneursh ip where a student's ability to 
borrow money may determine his or her chances of going into 
bus i ness . 
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Documenting the results of product use in detail or in 



assessing impact is expensive. For example, two interviewers 
spent four days interviewing participants at the six sites. It 
would have been better to have taken baseline measures in advance 
of introducing the R&D product. Decision options dependent on 
evaluative information about product use and/or impact need to be 
tightly structured to facilitate efficient and effective data 
collection. This should be done on a product-by-product, 
decision-by-decisioh basis. 

The twin forces of accountability and innovation management 
should drive investment of money in product use and/or impact 
studies. Assessment of entrepreneurship programs in West 
Virginia yielded a snapshot picture of activities in six school 
districts. If the districts are representative of others, then 
the information on numbers of students being served, teacher 
satisfaction with the materials, and so on could be used to 
(1) justify current expenditures of tiine and money for product 
development and distribution, or (2) establish baseline 
measurements for goals and priorities. Data-based management 
decisions are being used more and more by state departments of 
education. Product impact studies can determine changes that 
have occurred in vocational education programs as a result of R&D 
product use. This information can help establish a baseline of 
current practice in the field, thus influencing the need for new 
initiatives . 
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The primary implication from these Udies is that impact 
data can be collected from field site*^ using R&D products. The 
data becomes easier to collect the nearer it is to the source of 
product implementation. However# clues to resolution of the 
problem are more likely to come from studies of product use and 
impact; but these are the more expensive investigations to 
conduct • 

A second implication from these studies is the need to plan 
ahead. This is imperative for impact studies because baseline 
data should be collected. However, planning is essential for all 
studies: questions must be relevant to the purposes of the 
evaluationr and respondents should be key audiences who are in a 
position to resolve the problem(s) under investigation. 



48 



63 



if 



APPENDIX A 

CONCEPTUAL FRAMEWORK FOR ASSESSING IMPACT 



4 



ERIC 



64 



CONCEPTUAL FRAMEWORK FOR ASSRSSING IMPACT 

I ntroduct ion 

The conceptual framework for assessing the impact of R&D 
products may be found on the following page. It contains five 
stages related to product development and use. 

Criteria for assessing the impact of R&D products are 
arrayed within these five stages in figure A-1. Two different 
types of impact criteria are presented: formative and summative. 
Criteria inherent in the first four stages of the framework 
(development, dissemination, implementation, and utilization) are 
considered to be formative impact criteria. The primary purpose 
of evaluations conducted during these stages is to enhance the 
impact potential of the innovation. Criteria related to the last 
stage, effects, are considered summative impact criteria. 
Changes in people, organizations, or society are primary 
indicators of summative impact. Evidence of the permanence of 
the changes and unanticipated consequences of the innovation 
sometimes comes to light months or even years after the 
innovation has been introduced. 

Securing the funds, time, and interest needed to conduct 
impact assessments is difficult. Consequently, the authors 
advocate formative impact assessments as well as summative impact 
assessments. Formative impact assessments would examine the 
extent to which important criteria have been met at each stage of 
the R&D process. The effoct of an R&D product in meeting these 
criteria is cumulative. For example, a product that is system- 
atically developed, strategically disseminated, successfully 
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implemented, and used in an integrated manner is more likely to 
result in greater change than is a product not meeting these 
criteria. Of course, the impact of an R&D product also reflects 
the constraints and opportunities present in the adopting unit's 
environment. Therefore, a product developer or diL^seminator does 
not have total control over the changes that may or may not take 
place as a result of his or her product. 

\ 

\ 

Definitions of sixteen R&D impact criteria organized by the 
five program improvement stages follow. 

Development 

Impact begins with development of the product. Thus 
criteria used in the development process subsequently can be ased 
to select relevant research for developing other products, to 
encourage systematic testing and revision, and to upgrade product 
quality . 

Systematic development . A systematic process should be 
followed in developing innovations. An ideal process would 
include conducting research and needs assessment/task analysis; 
reviewing relev.it knowledge and practice; building a conceptual 
framework; sequencing development; conducting testing and 
revision cycles; disseminating the product; implementing the 
product; and evaluating the results. 

High quality . InnovationrD should reflect scholarship, be 
useful, communicate clearly, be marketable, and be free of 
biases. Content should be accurate, up-to-date, focused on 
essentials, and complete. 



User orientation . Representatives of relevant audiences 
should be identified and involved i • designing, testing, and 
using innovations. Primary audiences should receive priority in 
dissemination efforts. The resulting product should contain 
practical informatign organized in an easy-to-use format,. 

Distribution , 

The distribution criteria should encourage the spread of the 
RSiD product to primary audiences, increase the likelihood of the 
product's acceptance, and generate support for its. use. 

Strategic distribution . Cost-effective strategies for 
distributing an R&D product should be aevised on the basis of the 
characteristics of potential user's, site-specific factors, and 
features of the product itself. Distribution should reach 
opinion leaders and influential organizations in the external 

environment. . , 

Multiple channels . , More than one channel for conveying 
information about innovations should be used. Communication 
should include mass media (e.g., brochures sent out by mail) and 
interpersonal channels ( e .g . , technical assistance). Normally, 
information duplication and overlap are assets rather than 
liabilities during the distribution stage. 

Widespread distribution . Innovations should reach 
appropriate users. Thus, distribution to individuals in 
different roles, in diverse settings, and in .many geographic 
areas may be necessary. Secondary distribution through 
workshops, reprints, libraries, the ERIC system, and so on should 
be '-ncouraged. 
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Implementation 

Implementation strategies determine the product's poii of 
entry into an organization (e.g., at the classroom level). Cost 
feasibility studies and the need among practitioners for support 
systeij\s aid the timely implementation of R&D. 

Sequential implementation . The introduction of innovations 
should be sequenced to meet the needs and unique characteristics 
of an adopting site. Often potential users need to be introduced 
to segments of the innovation to avoid total rejection of the 
intervention. 

Support systems . Support systems necessary for encouraging 
the full use of an innovation should be operational at the time 
of implementation. These systems are of three types: personal 
resources (e.g., r " 'nistrative endorsement or site personnel 
endorsement), information resources (e.g., training in the use of 
support materials and procedures), and physical resources (e.g., 
dollars, supplies, and equipment). 

C ost feasibility . Information describing the innovation's 
resource requirements should allow quick and easy estimates of 
the costs likely to be incurred by an adopting unit. 

Use 

Various product use criteria can encourage the appropriate 
trial use of products, stimulate the integration of products into 
existing operations, and increase the chances of the continued 

use of products. 

M ultiple patterns of use . An innovation's use patterns will 

vary according to the conditions, intensity, level, frequency, 

t 
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and extent of use. The users' setting, role, and demographic 
characteristics create the conditions for different types of use, 
Multiple patterns of use and secondary use of RSeD by other than 
the primary user audience should be encouraged. 

Tine on task . An R&D product should be used frequently 
enough and long enough for its use to become an integral part of 
current practice. The audience's time in actually using the 
product should be maximized. 

Integrated use . The use of an innovative product should be 
intensive and pervasive throughout the organization. To accom-- 
plish this goal, personal commitment is required within the 
organization at all levels to institutionalize the product into 
organizational rout ines . 

Effects 

Product effects criteria should accurately describe changes 
in individuals, organizations, or society attributed , to use of 
R&D innovations. 

User sat isf act ion . The R&D product and its implementation 
should meet users' expectations and result in a positive user 
attitude toward the product. User satisfaction may be indicated 
by product advocacy or by creative adaptations. 

n di vidua 1 g rowt h . Innovations should contribute to changes 
in an individual's attitude, knowledge, or performance. 

Organizational change . R&D products should contribute to 
beneficial changes in the user's organizational policy, programs, 
practices, or structure. Beneficial changes may also include 
cosh and time savings over current practice. 
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Societal contributions . R&D products should contribute new 
and significant information with the potential to advance know- 
ledge, improve current practice, or influence social systems. 
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The Ohio Jnlverilty 



THE NATIONAL CENTER 
FOR RESEARCH IN VOCATIONAL EDUCATION 



March 14, 1983 



Address of 
State Director 



1960 Kenny Road 
Columbus, Ohio 43210 

Phone: 614-466-3655 

Cable: CTVOCEDOSU/Columbus, Ohio 



Dear : 

The National Center is designing an impact evaluation system 
of vocational education research and development. Such a 
system could strengthen coordination of program improvement 
activities as called for by Larry Selland at the 1982 
Dissemination and Utilization Conference. rhis work is 
supported by the U.S. Office of Vocational and Adult Education. 

We need support from your staff to build an effective systeia. 
This year, we plan to look primarily at the distribution of 
R&D products. Specifically, we need any available information 
on the number of R&D products distributed from program 
improvement projects completed in Fiscals Years 1981 and 1982. 
A member of my staff, William Hull, will be contacting a 
person in your state, , to obtain this information. 

Your state's participation is, of course, voluntary. A 
letter to the aforementioned person from Dr. Hull listing 
the specific R&D products will precede the telephone request. 
Please call him if you have questions about this activity. 

We appreciate your assistance and look forward to helpful 
information from your office. 

Sincerely , 



Robert E. Taylor 
Executive Director 

RET: jg 
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The Ohio State Unlvemity 



r FM RESEARCH IN VOCATIONAi EDUCATION 



April 18. 1983 



1960 Kenny Road 
Columbus, Ohio 43210 

Phone: 614-486-3655 

Cable: CTVOCEDOSU/Columbus, Ohio 



Dear Sirs: 

We are .compiling information on the distribution of research and development 
products from program improvement projects completed in Fiscal Years 1981 and 
1982. This work is supported by the U.S. Office of Vocational and Adult 
Education. Your help is needed to construct an evaluation system of vocational 
education research and development. The information requested should come from 
available records. Sending the information is completely voluntary. 

Please send the information on the attached sheet to me at the above address by 
May 10, 1983'. You can reach me at (800) 848-4815, ext. 517 if you hove quest- 
ions. You may wish to use the Computer-Based Message Switching System to send 
the information if you have a membership with Bibliographic Retrieval Services. 

'inclosed is a draft taxonomy of R&D product codes provided for your information, 
it was developed by Kay Adams and Debbie Bragg at the National Center. Please 
'eel free to use this information in any way that benefits your organization. 

We appreciate your assistance and look forward to communication from you. 
Sincerely, 



William L. Hull 
Research Specialist 

WLH:jg 

End osures 



Note: Please identify any products you have produced with federal funds 
(authorized by P.L. 94-482, sections 131, 132, 133) by title and 
publishing company that have been disseminated commercially during fiscal 
years 1981 or 1982. 



o 
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PRODUCT DISTRIBUTION RECORD: 1981 



Type of Product: 
Substantive Focus: 



Id, 

Sequence 

Prod, type 

Sub, Focus 

Mos. after 
release 



month 



year. 



1 . When was the pro act released? • 

2. How many copies of this product were distributed during the calendar ye^r FY 1981 
to the following audiences 



Role ^ 1 

^^0"^"^"^^^ Organization 


r*l 

1 

Total 


Number of Copies 


Administrative , 
Professionals and 
Support Personnel 


Teachers/ 
Trainers 


Students/ 
Trainees 


Education Orqanizations 

r^Hn A ♦* "1 r^n AClf*IlCieS 










olZaue t-ULlv^cl u iuii ^ 

Secondary schools, including 
district-wide audiences . 

Two-year community/technical 
institutes, proprietory 
schools, and programs for 
adults - - 


















Universities and other four-year 
inst itutions 










otliiT lui. lie Ortjanizat ions^ 

Military, correctional intJti- 
tutions, rehabilitation 
r;(^rvicoM, CKTA/JPTA proqramr>, 
aiKl community-based orcjnnl- 
zations such as industry 
onportunity centers 




\ 






other Private Sector Organizations 










BusinoFJS, industry, and labor 
councils 


1 

T0ta3. 


L,. — 









tr, ..iditxon to the copies indicated above, how many copies were sent to the followingi 
rersons, at the national or regional level 
persons in other countries 
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THANK YOU FOR VOLUNTARILY SHARING THIS INFORMATION 
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